The Effectiveness of <i>t</i>-Way Test Data Generation

Modern society is increasingly dependent on the correct functioning of software and increasingly so in areas that are considered safety related or safety critical. Therefore, there is an increasing need to be able to verify and validate that the software is in fact correct and will perform its intended function. Many approaches to this problem have been proposed; however, none seems likely to supplant the role of testing in the near future. If we accept that there is, and will be, a continuing need to be able to test software then the question becomes one of how can this be done effectively, both in terms of ability to detect errors and in terms of cost. One avenue of research that offers prospects of improving both of these aspects is the automatic generation of test data. There has recently been a large amount of work conducted in this area. One particularly promising direction has been the application of ideas from the field of experimental design and in particular, the field of t-way adequate factorial designs. The area however, is not without issues; there is evidence that the technique is capable of detecting errors but that evidence is not unequivocal. Moreover, as with almost all work in the area of automatic test generation, there has been very little comparative work comparing the technique with other test data generation techniques. Worse, there has been effectively no work done that compares any automatic test data generation technique with the effectiveness of tests generated by humans. Another major issue with the technique is the number of tests that applying the technique can result in. This implies that there is a need for an automated oracle if the technique is to be successfully applied. The flaw with this is of course that in most situations the oracle is the human that is conducting the tests, a point often ignored in testing research. The work presented here addresses both of these points. To do this I have used a code base taken from an industrial engine control system that has an existing set of high quality unit tests developed by hand. To complement this, several other techniques for automatically generating test data have been applied, namely random testing, random experimental designs and a technique for generating single factor experiments. To address the issue of being able to compare the error detection ability of all of the sets of test vectors, rather than the usual effectiveness surrogates of code coverage I have used mutation analysis on the code base to directly measure the ability of each set of test vectors to discover common coding errors. The results presented here show that test data generation techniques based on t-way factorial designs are at least as effective as handgenerated tests and superior to random testing and the factor experimental technique. The oracle problem associated with the factorial design techniques was addressed using a test set minimisation approach. The mutation tool monitored which vectors could “kill” which code mutants. After a subset of the test vectors had been run, the most effective vectors were retained and the rest discarded. Likewise, mutants that were killed were removed from further consideration and the process repeated. Experimental results show that this minimisation procedure is effective at reducing computational overhead and is capable of producing final sets of test vectors that are comparable in size with the sets of hand-generated tests and so amenable to final hand checking

The Effectiveness of t-way Test Data Generation

EThOS - Electronic Theses Online ServiceGBUnited Kingdo

The Csaw Mutation Tool Users Manual

Mutation is a technique that holds great promise in testing research, however it can be difficult getting hold of tools to allow the application of the technique to programs written in the C programming language. One tool that is available tool is Proteum. Here we describe another small tool set that can be used for performing mutation on C functions and provide guidance for the using the tool

The economics of unit testing

Conventional wisdom and anecdote suggests that testing takes between 30 to 50% of a project's effort. However testing is not a monolithic activity as it consists of a number of different phases such as unit testing, integration testing and finally system and acceptance test. Unit testing has received a lot of criticism in terms of the amount of time that it is perceived to take and its perceived costs. However it still remains an important verification activity being an effective means to test individual software components for boundary value behavior and ensure that all code has been exercised adequately. We examine the available data from three safety-related, industrial software projects that have made use of unit testing. Using this information we argue that the perceived costs of unit testing may be exaggerated and that the likely benefits in terms of defect detection are quite high in relation to those costs. We also discuss the different issues that have been found applying the technique at different phases of the development and using different methods to generate those tests. We also compare results we have obtained with empirical results from the literature and highlight some possible weakness of research in this area

Unit testing in practice

Unit testing is a technique that receives a lot of criticism in terms of the amount of time that it is perceived to take and in how much it costs to perform. However it is also the most effective means to test individual software components for boundary value behavior and ensure that all code has been exercise adequately (e.g. statement, branch or MC/DC coverage). In this paper we examine the available data from three safety related software projects undertaken by Pi Technology that have made use of unit testing. Additionally we discuss the different issues that have been found applying the technique at different phases of the development and using different methods to generate those test. In particular we provide an argument that the perceived costs of unit testing may be exaggerated and that the likely benefits in terms of defect detection are actually quite high in relation to those costs

AETG vs. Man: an Assessment of the Effectiveness of Combinatorial Test Data Generation

This paper reports on an industrial study of the effectiveness of test data generation. In the literature on the automatic generation of test data a number of techniques stand out as having received a significant amount of interest. One area that has achieved considerable attention is the use of combinatorial techniques to construct data adequate test sets that ensure all pairs, triples etc. of input variables are included in at least one test vector. There has been some systematic evaluation of the technique as applied to unit testing and, while there are indications that the technique can be effective, very little work has been performed using industrial code. Moreover, there has been no comparison of effectiveness of the technique for unit testing compared with tests that are generated by hand. In this paper we apply random and combinatorial (AETG) techniques to a number of functions drawn from industrial code with known faults and existing unit test suites. Results indicate that for simple cases combinatorial techniques can be as effective as the human-generated test, but there are instances associated with complex code where the technique performs poorly—but no worse than randomly generated data. Categories and Subject Descriptors D.2.5 [Testing and Debugging]- Testing tools- data generators

On Wheels, Nuts and Software

In this paper I examine the issue of whether we can trust software systems and put forward an argument that in an absolute sense the answer must necessarily be no for a multitude of reasons. The paper then examines the question of whether this is an actual impediment to the successful application of software with particular reference to automotive applications. In particular I examine the question of whether our expectations of software are compatible with the realities of road vehicle manufacture and use. I conclude that with good methodology and integration within a whole vehicle development process, software based systems can, and will provide levels of safety above those which are experienced today so long as certain critical constraints are met

AETG vs. Man: an Assessment of the Effectiveness of Combinatorial Test Data Generation

This paper reports on an industrial study of the effectiveness of test data generation. In the literature on the automatic generation of test data a number of techniques stand out as having received a significant amount of interest. One area that has achieved considerable attention is the use of combinatorial techniques to construct data adequate test sets that ensure all pairs, triples etc. of input variables are included in at least one test vector. There has been some systematic evaluation of the technique as applied to unit testing and, while there are indications that the technique can be effective, very little work has been performed using industrial code. Moreover, there has been no comparison of effectiveness of the technique for unit testing compared with tests that are generated by hand. In this paper we apply random and combinatorial (AETG) techniques to a number of functions drawn from industrial code with known faults and existing unit test suites. Results indicate that for simple cases combinatorial techniques can be as effective as the human-generated test, but there are instances associated with complex code where the technique performs poorly-but no worse than randomly generated data